0.1 BRC intro

The BRC is: Basic BRC plug.

1 Introduction to Data Wrangling with Tidy

1.1 A problem we face when dealing with data regularly.

Every dataset is different. Sometimes very different.

There are many ways to do things. Everyone has their favorite syntax.

The issue:
Many fundamental data processing functions exist in Base R and beyond. Sometimes they can be inconsistent or unnecessarily complex. The result is code that is confusing and doesn’t flow i.e. nested functions

1.2 What does it mean to be tidy?

Tidyverse is most importantly a philosophy for data analysis that more often then not makes wrangling data easier. The tidyverse community have built what they describe as an “opinionated” group of packages. These packages readily talk to one another.

  • More efficient code
  • Easier to remember syntax
  • Easier to read syntax

1.3 What does it actually mean to be tidy?

  • A defined vision for coding style in R
  • A defined vision for data formats in R
  • A defined vision for package design in R
  • Unified set of community pushing in a cohesive direction
  • Critical mass of poeple to influence the way the whole R commnity evolves

1.4 What are the main tools in the tidyverse?

  • readr – reading data into R
  • dplyr – manipulating data
  • tibble - working with tibbles
  • tidyr – miscellaneous tools for tidying data
  • ggplot2 – making pretty graphs
  • stringr – working with strings
  • purr - iterating over data
  • forcats - working with factors

Other tools have now been made for the tidy community. This community also overlaps with bioconductor. But the packages above are the linchpins that hold it together.

1.5 What are we doing today

Workflow Image for working with data.

2 Lets get tidy!

2.1 First step lets load in the data we are using today

2.2 Are all data frames equal?

##   salmon_id    common_name  age_classbylength length_mm IGF1_ng_ml
## 1     35032 Chinook salmon           yearling       147   41.26006
## 2     35035 Sockeye salmon           juvenile       121         NA
## 3     35036 Sockeye salmon           juvenile       112         NA
## 4     35037      Steelhead           juvenile       220   42.70981
## 5     35038      Steelhead           juvenile       152         NA
## 6     35033 Chinook salmon mixed age juvenile       444   62.11528
##   salmon_id    common_name  age_classbylength   variable     value
## 1     35032 Chinook salmon           yearling  length_mm 147.00000
## 2     35032 Chinook salmon           yearling IGF1_ng_ml  41.26006
## 3     35033 Chinook salmon mixed age juvenile  length_mm 444.00000
## 4     35033 Chinook salmon mixed age juvenile IGF1_ng_ml  62.11528
## 5     35034 Sockeye salmon           juvenile  length_mm 139.00000
## 6     35034 Sockeye salmon           juvenile IGF1_ng_ml        NA
##   salmon_id    common_name  age_classbylength  variable value
## 1     35032 Chinook salmon           yearling length_mm   147
## 2     35033 Chinook salmon mixed age juvenile length_mm   444
## 3     35034 Sockeye salmon           juvenile length_mm   139
## 4     35035 Sockeye salmon           juvenile length_mm   121
## 5     35036 Sockeye salmon           juvenile length_mm   112
## 6     35037      Steelhead           juvenile length_mm   220
##   salmon_id    common_name  age_classbylength   variable    value
## 1     35032 Chinook salmon           yearling IGF1_ng_ml 41.26006
## 2     35033 Chinook salmon mixed age juvenile IGF1_ng_ml 62.11528
## 3     35034 Sockeye salmon           juvenile IGF1_ng_ml       NA
## 4     35035 Sockeye salmon           juvenile IGF1_ng_ml       NA
## 5     35036 Sockeye salmon           juvenile IGF1_ng_ml       NA
## 6     35037      Steelhead           juvenile IGF1_ng_ml 42.70981

2.3 What is a tidy dataset?

A tidy dataset is a data frame (or table) for which the following are true:

  • Each variable has its own column
  • Each observation has its own row
  • Each value has its own cell

Our first dataframe is tidy

2.4 Why bother?

Consistent dataframe layouts help to ensure that all values are present and that relationships between data points are clear.

R is a vectorized programming language. R builds dataframes from vectotrs, and R works best when its operation are vectorized. Tidy data utilizes of both of these aspects of R.

=> Precise and Fast

2.5 Lets load in the tidyverse

## ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
## ✔ ggplot2 3.2.1     ✔ purrr   0.3.3
## ✔ tibble  2.1.3     ✔ dplyr   0.8.3
## ✔ tidyr   1.0.0     ✔ stringr 1.4.0
## ✔ readr   1.3.1     ✔ forcats 0.4.0
## ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()

3 #These tools hav same logic

-> insert graphic here https://github.com/trinker/tidyr_in_a_nutshell

4 dplyr: A tool to access and manipulate data in a dataframe

4.1 Select

Select allows you to make a vector or dataframe from a specific variable or variables

## # A tibble: 97 x 1
##    common_name   
##    <chr>         
##  1 Chinook salmon
##  2 Sockeye salmon
##  3 Sockeye salmon
##  4 Steelhead     
##  5 Steelhead     
##  6 Chinook salmon
##  7 Sockeye salmon
##  8 Steelhead     
##  9 Steelhead     
## 10 Steelhead     
## # … with 87 more rows
## # A tibble: 97 x 2
##    age_classbylength  common_name   
##    <chr>              <chr>         
##  1 yearling           Chinook salmon
##  2 juvenile           Sockeye salmon
##  3 juvenile           Sockeye salmon
##  4 juvenile           Steelhead     
##  5 juvenile           Steelhead     
##  6 mixed age juvenile Chinook salmon
##  7 juvenile           Sockeye salmon
##  8 juvenile           Steelhead     
##  9 juvenile           Steelhead     
## 10 juvenile           Steelhead     
## # … with 87 more rows
## # A tibble: 97 x 4
##    salmon_id common_name    age_classbylength  IGF1_ng_ml
##        <dbl> <chr>          <chr>                   <dbl>
##  1     35032 Chinook salmon yearling                 41.3
##  2     35035 Sockeye salmon juvenile                 NA  
##  3     35036 Sockeye salmon juvenile                 NA  
##  4     35037 Steelhead      juvenile                 42.7
##  5     35038 Steelhead      juvenile                 NA  
##  6     35033 Chinook salmon mixed age juvenile       62.1
##  7     35034 Sockeye salmon juvenile                 NA  
##  8     35048 Steelhead      juvenile                 24.2
##  9     35049 Steelhead      juvenile                 NA  
## 10     35050 Steelhead      juvenile                 63.5
## # … with 87 more rows
## # A tibble: 97 x 3
##    common_name    age_classbylength  length_mm
##    <chr>          <chr>                  <dbl>
##  1 Chinook salmon yearling                 147
##  2 Sockeye salmon juvenile                 121
##  3 Sockeye salmon juvenile                 112
##  4 Steelhead      juvenile                 220
##  5 Steelhead      juvenile                 152
##  6 Chinook salmon mixed age juvenile       444
##  7 Sockeye salmon juvenile                 139
##  8 Steelhead      juvenile                 288
##  9 Steelhead      juvenile                 190
## 10 Steelhead      juvenile                 283
## # … with 87 more rows

4.2 Filter

Filter allows you to access observations based on specific criteria

## # A tibble: 11 x 5
##    salmon_id common_name    age_classbylength length_mm IGF1_ng_ml
##        <dbl> <chr>          <chr>                 <dbl>      <dbl>
##  1     35035 Sockeye salmon juvenile                121         NA
##  2     35036 Sockeye salmon juvenile                112         NA
##  3     35034 Sockeye salmon juvenile                139         NA
##  4     35144 Sockeye salmon juvenile                140         NA
##  5     35147 Sockeye salmon juvenile                115         NA
##  6     35096 Sockeye salmon juvenile                115         NA
##  7     35097 Sockeye salmon juvenile                110         NA
##  8     35098 Sockeye salmon juvenile                112         NA
##  9     35099 Sockeye salmon juvenile                111         NA
## 10     35100 Sockeye salmon juvenile                118         NA
## 11     35119 Sockeye salmon juvenile                122         NA
## # A tibble: 57 x 5
##    salmon_id common_name    age_classbylength  length_mm IGF1_ng_ml
##        <dbl> <chr>          <chr>                  <dbl>      <dbl>
##  1     35032 Chinook salmon yearling                 147       41.3
##  2     35035 Sockeye salmon juvenile                 121       NA  
##  3     35036 Sockeye salmon juvenile                 112       NA  
##  4     35033 Chinook salmon mixed age juvenile       444       62.1
##  5     35034 Sockeye salmon juvenile                 139       NA  
##  6     35142 Chinook salmon yearling                 149       66.5
##  7     35143 Chinook salmon yearling                 204       80.9
##  8     35144 Sockeye salmon juvenile                 140       NA  
##  9     35145 Chinook salmon yearling                 130       23.4
## 10     35146 Chinook salmon mixed age juvenile       422      101. 
## # … with 47 more rows
## # A tibble: 59 x 5
##    salmon_id common_name    age_classbylength  length_mm IGF1_ng_ml
##        <dbl> <chr>          <chr>                  <dbl>      <dbl>
##  1     35032 Chinook salmon yearling                 147       41.3
##  2     35035 Sockeye salmon juvenile                 121       NA  
##  3     35036 Sockeye salmon juvenile                 112       NA  
##  4     35033 Chinook salmon mixed age juvenile       444       62.1
##  5     35034 Sockeye salmon juvenile                 139       NA  
##  6     35142 Chinook salmon yearling                 149       66.5
##  7     35143 Chinook salmon yearling                 204       80.9
##  8     35144 Sockeye salmon juvenile                 140       NA  
##  9     35145 Chinook salmon yearling                 130       23.4
## 10     35146 Chinook salmon mixed age juvenile       422      101. 
## # … with 49 more rows
## # A tibble: 36 x 5
##    salmon_id common_name    age_classbylength  length_mm IGF1_ng_ml
##        <dbl> <chr>          <chr>                  <dbl>      <dbl>
##  1     35036 Sockeye salmon juvenile                 112       NA  
##  2     35037 Steelhead      juvenile                 220       42.7
##  3     35033 Chinook salmon mixed age juvenile       444       62.1
##  4     35048 Steelhead      juvenile                 288       24.2
##  5     35050 Steelhead      juvenile                 283       63.5
##  6     35051 Steelhead      juvenile                 279       61.2
##  7     35052 Steelhead      juvenile                 235       30.6
##  8     35053 Steelhead      juvenile                 230       49.4
##  9     35056 Steelhead      juvenile                 208       57.4
## 10     35057 Steelhead      juvenile                 240       20.2
## # … with 26 more rows

4.3 Arrange

Arrange sorts the dataframe based on a specific variable or variables

## # A tibble: 97 x 5
##    salmon_id common_name    age_classbylength length_mm IGF1_ng_ml
##        <dbl> <chr>          <chr>                 <dbl>      <dbl>
##  1     35095 Chinook salmon subyearling              90         NA
##  2     35097 Sockeye salmon juvenile                110         NA
##  3     35099 Sockeye salmon juvenile                111         NA
##  4     35036 Sockeye salmon juvenile                112         NA
##  5     35098 Sockeye salmon juvenile                112         NA
##  6     35147 Sockeye salmon juvenile                115         NA
##  7     35096 Sockeye salmon juvenile                115         NA
##  8     35100 Sockeye salmon juvenile                118         NA
##  9     35035 Sockeye salmon juvenile                121         NA
## 10     35119 Sockeye salmon juvenile                122         NA
## # … with 87 more rows
## # A tibble: 97 x 5
##    salmon_id common_name    age_classbylength  length_mm IGF1_ng_ml
##        <dbl> <chr>          <chr>                  <dbl>      <dbl>
##  1     35033 Chinook salmon mixed age juvenile       444      62.1 
##  2     35146 Chinook salmon mixed age juvenile       422     101.  
##  3     35110 Chinook salmon mixed age juvenile       275      81.5 
##  4     35129 Chinook salmon yearling                 225      72.7 
##  5     35103 Chinook salmon yearling                 216      81.2 
##  6     35115 Chinook salmon yearling                 215      53.5 
##  7     35112 Chinook salmon yearling                 205      90.5 
##  8     35143 Chinook salmon yearling                 204      80.9 
##  9     35079 Chinook salmon yearling                 199      53.2 
## 10     35081 Chinook salmon yearling                 196       5.56
## # … with 87 more rows

4.4 Mutate

Mutate creates a new variable based on some form of computation

## # A tibble: 97 x 6
##    salmon_id common_name  age_classbyleng… length_mm IGF1_ng_ml `scale(IGF1_ng_…
##        <dbl> <chr>        <chr>                <dbl>      <dbl>            <dbl>
##  1     35032 Chinook sal… yearling               147       41.3           -0.258
##  2     35035 Sockeye sal… juvenile               121       NA             NA    
##  3     35036 Sockeye sal… juvenile               112       NA             NA    
##  4     35037 Steelhead    juvenile               220       42.7           -0.191
##  5     35038 Steelhead    juvenile               152       NA             NA    
##  6     35033 Chinook sal… mixed age juven…       444       62.1            0.704
##  7     35034 Sockeye sal… juvenile               139       NA             NA    
##  8     35048 Steelhead    juvenile               288       24.2           -1.04 
##  9     35049 Steelhead    juvenile               190       NA             NA    
## 10     35050 Steelhead    juvenile               283       63.5            0.766
## # … with 87 more rows
## # A tibble: 97 x 6
##    salmon_id common_name  age_classbyleng… length_mm IGF1_ng_ml IGFngml_zscore[…
##        <dbl> <chr>        <chr>                <dbl>      <dbl>            <dbl>
##  1     35032 Chinook sal… yearling               147       41.3           -0.258
##  2     35035 Sockeye sal… juvenile               121       NA             NA    
##  3     35036 Sockeye sal… juvenile               112       NA             NA    
##  4     35037 Steelhead    juvenile               220       42.7           -0.191
##  5     35038 Steelhead    juvenile               152       NA             NA    
##  6     35033 Chinook sal… mixed age juven…       444       62.1            0.704
##  7     35034 Sockeye sal… juvenile               139       NA             NA    
##  8     35048 Steelhead    juvenile               288       24.2           -1.04 
##  9     35049 Steelhead    juvenile               190       NA             NA    
## 10     35050 Steelhead    juvenile               283       63.5            0.766
## # … with 87 more rows

4.5 Summarize

Summarize applies aggregating or summary function to a group

## # A tibble: 4 x 2
##   common_name    count
##   <chr>          <int>
## 1 Chinook salmon    46
## 2 Coho salmon        2
## 3 Sockeye salmon    11
## 4 Steelhead         38
## # A tibble: 4 x 2
##   common_name    IGF1_ng_ml_ave
##   <chr>                   <dbl>
## 1 Chinook salmon           46.8
## 2 Coho salmon              73.6
## 3 Sockeye salmon          NaN  
## 4 Steelhead                46.1

4.6 Group

Grouping can also help ask questions with other fucntions

## # A tibble: 8 x 5
## # Groups:   common_name [4]
##   salmon_id common_name    age_classbylength length_mm IGF1_ng_ml
##       <dbl> <chr>          <chr>                 <dbl>      <dbl>
## 1     35038 Steelhead      juvenile                152       NA  
## 2     35055 Steelhead      juvenile                123       55.7
## 3     35145 Chinook salmon yearling                130       23.4
## 4     35085 Coho salmon    yearling                140       NA  
## 5     35087 Coho salmon    yearling                164       73.6
## 6     35095 Chinook salmon subyearling              90       NA  
## 7     35097 Sockeye salmon juvenile                110       NA  
## 8     35099 Sockeye salmon juvenile                111       NA
## # A tibble: 95 x 5
## # Groups:   common_name [3]
##    salmon_id common_name    age_classbylength  length_mm IGF1_ng_ml
##        <dbl> <chr>          <chr>                  <dbl>      <dbl>
##  1     35032 Chinook salmon yearling                 147       41.3
##  2     35035 Sockeye salmon juvenile                 121       NA  
##  3     35036 Sockeye salmon juvenile                 112       NA  
##  4     35037 Steelhead      juvenile                 220       42.7
##  5     35038 Steelhead      juvenile                 152       NA  
##  6     35033 Chinook salmon mixed age juvenile       444       62.1
##  7     35034 Sockeye salmon juvenile                 139       NA  
##  8     35048 Steelhead      juvenile                 288       24.2
##  9     35049 Steelhead      juvenile                 190       NA  
## 10     35050 Steelhead      juvenile                 283       63.5
## # … with 85 more rows
## # A tibble: 97 x 6
## # Groups:   common_name [4]
##    salmon_id common_name   age_classbylength length_mm IGF1_ng_ml IGFngml_zscore
##        <dbl> <chr>         <chr>                 <dbl>      <dbl>          <dbl>
##  1     35032 Chinook salm… yearling                147       41.3         -0.236
##  2     35035 Sockeye salm… juvenile                121       NA           NA    
##  3     35036 Sockeye salm… juvenile                112       NA           NA    
##  4     35037 Steelhead     juvenile                220       42.7         -0.176
##  5     35038 Steelhead     juvenile                152       NA           NA    
##  6     35033 Chinook salm… mixed age juveni…       444       62.1          0.652
##  7     35034 Sockeye salm… juvenile                139       NA           NA    
##  8     35048 Steelhead     juvenile                288       24.2         -1.12 
##  9     35049 Steelhead     juvenile                190       NA           NA    
## 10     35050 Steelhead     juvenile                283       63.5          0.890
## # … with 87 more rows

5 Piping (%>%): A way to string together functions together

Piping allows you to pass the result from one expression directly into another.

-> same graphic as before , but extend https://github.com/trinker/tidyr_in_a_nutshell

5.1 Piping versus not piping

## # A tibble: 4 x 2
##   common_name    IGF1_ng_ml_ave
##   <chr>                   <dbl>
## 1 Chinook salmon           46.8
## 2 Coho salmon              73.6
## 3 Sockeye salmon          NaN  
## 4 Steelhead                46.1
## # A tibble: 4 x 2
##   common_name    IGF1_ng_ml_ave
##   <chr>                   <dbl>
## 1 Chinook salmon           46.8
## 2 Coho salmon              73.6
## 3 Sockeye salmon          NaN  
## 4 Steelhead                46.1

5.2 Building pipes together

## # A tibble: 2 x 2
##   common_name    IGF1_ng_ml_ave
##   <chr>                   <dbl>
## 1 Chinook salmon           77.9
## 2 Steelhead                45.3
## # A tibble: 6 x 3
## # Groups:   common_name [4]
##   common_name    size       IGF1_ng_ml_ave
##   <chr>          <chr>               <dbl>
## 1 Chinook salmon big_fish             77.9
## 2 Chinook salmon small_fish           39.5
## 3 Coho salmon    small_fish           73.6
## 4 Sockeye salmon small_fish          NaN  
## 5 Steelhead      big_fish             45.3
## 6 Steelhead      small_fish           47.3
## # A tibble: 4 x 3
## # Groups:   common_name [2]
##   common_name    size       IGF1_ng_ml_ave
##   <chr>          <chr>               <dbl>
## 1 Chinook salmon big_fish             77.9
## 2 Chinook salmon small_fish           39.5
## 3 Steelhead      big_fish             45.3
## 4 Steelhead      small_fish           47.3

5.3 Say want to graph what is going on with these fish? age species. IGF per length.

CHALLENGE

ANSWER

5.4 So we blasted through what being tidy can give you. So how we do get tidy? Reading the data in and making tibbles.

Now lets tidy some data. We know how to work a little bit with some data.

ReadR: read_csv(): comma separated (CSV) files read_tsv(): tab separated files read_delim(): general delimited files read_fwf(): fixed width files read_table(): tabular files where columns are separated by white-space. read_log(): web log files

## Parsed with column specification:
## cols(
##   ENTREZ = col_double(),
##   CD34_1 = col_double(),
##   ORTHO_1 = col_double(),
##   CD34_2 = col_double(),
##   ORTHO_2 = col_double()
## )

#subsetting tibbles

## # A tibble: 100 x 1
##    ENTREZ
##    <chr> 
##  1 350   
##  2 351   
##  3 353   
##  4 354   
##  5 355   
##  6 356   
##  7 357   
##  8 358   
##  9 359   
## 10 360   
## # … with 90 more rows
## # A tibble: 1 x 5
##   ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
##   <chr>   <int>   <int>  <int>   <int>
## 1 350       204       0    103       0
## # A tibble: 100 x 1
##    ENTREZ
##    <chr> 
##  1 350   
##  2 351   
##  3 353   
##  4 354   
##  5 355   
##  6 356   
##  7 357   
##  8 358   
##  9 359   
## 10 360   
## # … with 90 more rows
##   [1] "350" "351" "353" "354" "355" "356" "357" "358" "359" "360" "361" "362"
##  [13] "363" "364" "366" "367" "368" "369" "372" "373" "374" "375" "377" "378"
##  [25] "379" "381" "382" "383" "384" "387" "388" "389" "390" "391" "392" "393"
##  [37] "394" "395" "396" "397" "398" "399" "400" "401" "402" "403" "405" "406"
##  [49] "407" "408" "409" "410" "411" "412" "414" "415" "416" "417" "419" "420"
##  [61] "421" "427" "429" "430" "432" "433" "434" "435" "440" "443" "444" "445"
##  [73] "460" "462" "463" "466" "467" "468" "471" "472" "473" "474" "475" "476"
##  [85] "477" "478" "479" "480" "481" "482" "483" "486" "487" "488" "489" "490"
##  [97] "491" "492" "493" "495"
##   [1] "350" "351" "353" "354" "355" "356" "357" "358" "359" "360" "361" "362"
##  [13] "363" "364" "366" "367" "368" "369" "372" "373" "374" "375" "377" "378"
##  [25] "379" "381" "382" "383" "384" "387" "388" "389" "390" "391" "392" "393"
##  [37] "394" "395" "396" "397" "398" "399" "400" "401" "402" "403" "405" "406"
##  [49] "407" "408" "409" "410" "411" "412" "414" "415" "416" "417" "419" "420"
##  [61] "421" "427" "429" "430" "432" "433" "434" "435" "440" "443" "444" "445"
##  [73] "460" "462" "463" "466" "467" "468" "471" "472" "473" "474" "475" "476"
##  [85] "477" "478" "479" "480" "481" "482" "483" "486" "487" "488" "489" "490"
##  [97] "491" "492" "493" "495"

#Converting Tibbles - Back and Forth

## # A tibble: 100 x 5
##    ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
##     <int>  <int>   <int>  <int>   <int>
##  1    350    204       0    103       0
##  2    351  15586     479  10476      39
##  3    353    842     355   1188      86
##  4    354      0       0      0       0
##  5    355    123     291    139      16
##  6    356      1       1      0       0
##  7    357    380       3    177       0
##  8    358    572    2225    597    4051
##  9    359      0      12      1       0
## 10    360    320     502     46    1114
## # … with 90 more rows
## # A tibble: 100 x 5
##    ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
##    <chr>   <int>   <int>  <int>   <int>
##  1 350       204       0    103       0
##  2 351     15586     479  10476      39
##  3 353       842     355   1188      86
##  4 354         0       0      0       0
##  5 355       123     291    139      16
##  6 356         1       1      0       0
##  7 357       380       3    177       0
##  8 358       572    2225    597    4051
##  9 359         0      12      1       0
## 10 360       320     502     46    1114
## # … with 90 more rows
##     ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
## 1      350    204       0    103       0
## 2      351  15586     479  10476      39
## 3      353    842     355   1188      86
## 4      354      0       0      0       0
## 5      355    123     291    139      16
## 6      356      1       1      0       0
## 7      357    380       3    177       0
## 8      358    572    2225    597    4051
## 9      359      0      12      1       0
## 10     360    320     502     46    1114
## 11     361      0       1      0       0
## 12     362      3       1     15       0
## 13     363     14       6      4       1
## 14     364      7       0      1       0
## 15     366      6       0      1       0
## 16     367     42       0     51       1
## 17     368     28       0     24       0
## 18     369   1204    1034    833     478
## 19     372   2829    1864   2741     771
## 20     373    179     728    148     795
## 21     374     76       5    138       2
## 22     375   4428    6697   4970    4328
## 23     377   3170     314   2576      11
## 24     378   1839    4845   1767    2975
## 25     379    178       1    181       0
## 26     381   1617     574   1159     339
## 27     382   2874    2265   1746    1668
## 28     383     63    1632     40     721
## 29     384    148   10977    118      94
## 30     387   8899    2457   7405    1228
## 31     388  12598     171   5090      70
## 32     389   2709     193   2313       5
## 33     390   1004       0    395       0
## 34     391   1038     577   1176     164
## 35     392   1527     304    786      71
## 36     393   2949     138   1540       3
## 37     394   1525     464   1062     134
## 38     395    348      67    123       0
## 39     396   6503     702   4723     169
## 40     397  12997     410  11265      38
## 41     398      0       0      0       0
## 42     399    223      11    422       2
## 43     400   1188     147    806      56
## 44     401      0       0      0       0
## 45     402    504     218    496      80
## 46     403    289      25    166       4
## 47     405   1481     824   1004     812
## 48     406    295     175     87      35
## 49     407      4       1      2       1
## 50     408   2451     111   1523       6
## 51     409   2480    1819   1356     226
## 52     410    433     197    215      77
## 53     411    829     217    441     131
## 54     412    312      45    138      17
## 55     414    516      15    396       9
## 56     415     20       0     13       0
## 57     416      2       1      4       0
## 58     417      0       0      0       0
## 59     419      6       5      2       7
## 60     420    141    1136     94    1217
## 61     421    213     255     93     208
## 62     427   4699   11889   1729     926
## 63     429      0       0      2       0
## 64     430     34      13     22       0
## 65     432     63       0     55       1
## 66     433     38       0     26       0
## 67     434      1       1      2       0
## 68     435    408     151    284      34
## 69     440    157    2520    111     535
## 70     443      5       0      4       0
## 71     444   1583     151    747      14
## 72     445    116      15     90       1
## 73     460     34       0     68       0
## 74     462     70       0     31       1
## 75     463   1244     118    492      71
## 76     466    538     480    393     218
## 77     467   2506     402   2130      18
## 78     468   7991    6132   5307    1883
## 79     471   1272     771   1392      53
## 80     472   1389     628    739     138
## 81     473   4173     783   1901     776
## 82     474      0       0      0       0
## 83     475    284      83    467      34
## 84     476   4952    3453   4202    1416
## 85     477     13       3     26       2
## 86     478     78     121     67       0
## 87     479     17       0      4       0
## 88     480      9       5     11       1
## 89     481   1937      18   1017      34
## 90     482    157    1392     75    1660
## 91     483   1075    1454   1789    1141
## 92     486     47       0     18       0
## 93     487     29      33     19       3
## 94     488   4529    1118   2925     269
## 95     489   3465     153   3188       8
## 96     490   1610    1263    913     665
## 97     491     12       1      4       0
## 98     492      4       0      6       0
## 99     493   5011    3585   3053     743
## 100    495      0       0      0       0

6 Make your own tibble. Lets grab some metadata

## Loading required package: AnnotationDbi
## Warning: package 'AnnotationDbi' was built under R version 3.6.1
## Loading required package: stats4
## Loading required package: BiocGenerics
## Loading required package: parallel
## 
## Attaching package: 'BiocGenerics'
## The following objects are masked from 'package:parallel':
## 
##     clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
##     clusterExport, clusterMap, parApply, parCapply, parLapply,
##     parLapplyLB, parRapply, parSapply, parSapplyLB
## The following objects are masked from 'package:dplyr':
## 
##     combine, intersect, setdiff, union
## The following objects are masked from 'package:stats':
## 
##     IQR, mad, sd, var, xtabs
## The following objects are masked from 'package:base':
## 
##     anyDuplicated, append, as.data.frame, basename, cbind, colnames,
##     dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
##     grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
##     order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
##     rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
##     union, unique, unsplit, which, which.max, which.min
## Loading required package: Biobase
## Welcome to Bioconductor
## 
##     Vignettes contain introductory material; view with
##     'browseVignettes()'. To cite Bioconductor, see
##     'citation("Biobase")', and for packages 'citation("pkgname")'.
## Loading required package: IRanges
## Warning: package 'IRanges' was built under R version 3.6.1
## Loading required package: S4Vectors
## Warning: package 'S4Vectors' was built under R version 3.6.1
## 
## Attaching package: 'S4Vectors'
## The following object is masked from 'package:plotly':
## 
##     rename
## The following objects are masked from 'package:dplyr':
## 
##     first, rename
## The following object is masked from 'package:tidyr':
## 
##     expand
## The following object is masked from 'package:base':
## 
##     expand.grid
## 
## Attaching package: 'IRanges'
## The following object is masked from 'package:plotly':
## 
##     slice
## The following objects are masked from 'package:dplyr':
## 
##     collapse, desc, slice
## The following object is masked from 'package:purrr':
## 
##     reduce
## 
## Attaching package: 'AnnotationDbi'
## The following object is masked from 'package:plotly':
## 
##     select
## The following object is masked from 'package:dplyr':
## 
##     select
## 
## Loading required package: GenomicFeatures
## Loading required package: GenomeInfoDb
## Loading required package: GenomicRanges
## Warning: package 'GenomicRanges' was built under R version 3.6.1
## 'select()' returned 1:1 mapping between keys and columns
## 'select()' returned 1:1 mapping between keys and columns

7 What is wrong with this dataframe from a tidy viewpoint?

  • Each variable has its own column
  • Each observation has its own row
  • Each value has its own cell
## # A tibble: 100 x 5
##    ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
##    <chr>   <int>   <int>  <int>   <int>
##  1 350       204       0    103       0
##  2 351     15586     479  10476      39
##  3 353       842     355   1188      86
##  4 354         0       0      0       0
##  5 355       123     291    139      16
##  6 356         1       1      0       0
##  7 357       380       3    177       0
##  8 358       572    2225    597    4051
##  9 359         0      12      1       0
## 10 360       320     502     46    1114
## # … with 90 more rows

A single variable with multiple columns

8 How do we get tidy? - Pivot tools (formerly known as gather/spread )

## # A tibble: 100 x 5
##    ENTREZ CD34_1 ORTHO_1 CD34_2 ORTHO_2
##    <chr>   <int>   <int>  <int>   <int>
##  1 350       204       0    103       0
##  2 351     15586     479  10476      39
##  3 353       842     355   1188      86
##  4 354         0       0      0       0
##  5 355       123     291    139      16
##  6 356         1       1      0       0
##  7 357       380       3    177       0
##  8 358       572    2225    597    4051
##  9 359         0      12      1       0
## 10 360       320     502     46    1114
## # … with 90 more rows

9 What next?

## # A tibble: 400 x 3
##    ENTREZ Sample  counts
##    <chr>  <chr>    <int>
##  1 350    CD34_1     204
##  2 350    ORTHO_1      0
##  3 350    CD34_2     103
##  4 350    ORTHO_2      0
##  5 351    CD34_1   15586
##  6 351    ORTHO_1    479
##  7 351    CD34_2   10476
##  8 351    ORTHO_2     39
##  9 353    CD34_1     842
## 10 353    ORTHO_1    355
## # … with 390 more rows

10 Multiple variables in a single column

How do we get tidy? - Cleaning up

## # A tibble: 400 x 5
##    ENTREZ Sample  CellType Rep   counts
##    <chr>  <chr>   <chr>    <chr>  <int>
##  1 350    CD34_1  CD34     1        204
##  2 350    ORTHO_1 ORTHO    1          0
##  3 350    CD34_2  CD34     2        103
##  4 350    ORTHO_2 ORTHO    2          0
##  5 351    CD34_1  CD34     1      15586
##  6 351    ORTHO_1 ORTHO    1        479
##  7 351    CD34_2  CD34     2      10476
##  8 351    ORTHO_2 ORTHO    2         39
##  9 353    CD34_1  CD34     1        842
## 10 353    ORTHO_1 ORTHO    1        355
## # … with 390 more rows

12 At this point we have covered the most essential facets of tidy

readr – reading data into R dplyr – mainpulating data tibble - working with tibbles tidyr – miscellaneous tools for tidying data ggplot2 – making pretty graphs stringr – working with strings purr - iterating over data forcats - working with factors

#stringr

Character manipulation: these functions allow you to manipulate individual characters within the strings in character vectors.

Whitespace tools to add, remove, and manipulate whitespace.

Locale sensitive operations whose operations will vary from locale to locale.

Pattern matching functions. These recognise four engines of pattern description. The most common is regular expressions, but there are three other tools.

#purr

applying functions to datasets

13 tidy history and ethos

Hadley Wickham (Chief Scientist at RStudio) has been the driving force behind the tidyverse. It now serves as the basis for many R packages and functions.

There is a lot of support at https://www.tidyverse.org/

Hadley also wrote a paper about why tidy data is best: www.jstatsoft.org/v59/i10/paper.

13.1 BRC outro